1. Technical Field
The present invention relates to a road marking image processing device, a road marking image processing method and a program thereof, which acquires an image of a road surface by using a camera loaded on a vehicle and recognizes a road marking within the image. More specifically, the present invention relates to a road marking image processing device, a road marking image processing method and a program thereof, which performs image processing by converting an image picked up by a camera into a bird's-eye image.
2. Related Art
As a safety measure for automobiles, there is known a technique which supports driving of an automobile based on an image of areas in the vicinity of the vehicle picked up by an onboard camera. For example, there is a system which specifies the position of a vehicle, e.g., precisely specifies the traffic lane where the vehicle is traveling, by capturing an image of a traveling road surface of the vehicle by using the onboard camera and recognizing a road marking drawn on the road surface from the captured image.
As a technique for processing an image captured by an onboard camera, Patent Document 1 discloses a device which converts an input image from an onboard camera into a bird's-eye image, and creates a wide-range bird's-eye image by synthesizing the bird's-eye images acquired in order. The device disclosed in Patent Document 1 sets a conversion boundary line for the image from the onboard camera, which is a captured image of the traveling path ahead the vehicle, converts the section closer to the own car than the conversion boundary line into a bird's-eye image, synthesizes n-pieces of the bird's-eye images, and displays the synthesized bird's-eye image.
Further, as other related techniques, there are also techniques disclosed in Patent Document 2 and Patent Document 3. The technique disclosed in Patent Document 2 is a technique which detects a shift between a first bird's-eye image and a second bird's-eye image capture by an imaging module, adds an image of the shift part to the second bird's-eye image, and displays an image containing a region out of the current visual field of the imaging module. The technique disclosed in Patent Document 3 is a system which converts an image captured by a single onboard camera into a bird's-eye image, and creates and displays a peripheral image of a vehicle by connecting the acquired bird's-eye images according to chronological changes.    Patent Document 1: Japanese Patent Publication No. 3847547    Patent Document 2: Japanese Patent Publication No. 3521859    Patent Document 3: Japanese Unexamined Patent Publication 2007-96497
However, the techniques disclosed in Patent Documents 1-3 are designed on assumption that a highly precise camera parameter can be acquired. Thus, when it is tried to acquire a synthesized bird's-eye image by converting and synthesizing the images acquired through capturing images of a road surface moment by moment with a camera loaded on a vehicle, it is difficult to acquire a fine synthesized bird's-eye image since there is actually an influence of a camera parameter error when converting the captured images to the bird's-eye images. Further, it is also difficult to acquire a fine synthesized bird's-eye image due to an influence of noises, when noises caused by reflection of light of other vehicles from the road surface, extraneous matters attached to the camera lens, and a shadow of own vehicle and the like are captured into the image.
For example, regarding the camera parameter, the shift and distortion on the bird's-eye images become more extensive as the distance from the camera becomes farther if the accuracy of the attaching angle is insufficiently secured when attaching the onboard camera to the vehicle. This will be described by referring to the drawings.
FIG. 6 is an illustration for describing a placing state of a camera 602 that is placed on a rear part of a vehicle 601. It is assumed that the height from a ground to the camera 602 is h, and the angle between the optical axis of the camera 602 and the vertical direction is θ. FIG. 18 is a table showing the relationship between the angle from the camera optical axis and the horizontal distance from the point on the ground in that angle direction to the camera 602 when the attaching angle of the camera is set in three kinds under a camera setting environment as shown in FIG. 6. Here, it is assumed that the camera 602 has no lens distortion for simplifying the explanation, and illustrated are the cases where the height h from the ground to the camera 602 is 80 cm and the angle θ between the optical axis of the camera 602 and the vertical direction is set in three kinds such as 30 degrees, 31 degrees, and 33 degrees.
As shown in FIG. 18, when the angle θ supposed in the specification is 30 degrees but the camera 602 is actually attached at the angle θ of 31 degrees or 33 degrees, for example, it can be seen that the error in the horizontal distance between the point on the ground in the 0-degree angle direction from the optical axis of the camera 602 and the camera 602 is about several cm such as 0.02 m (=0.48 m−0.46 m) and 0.06 m (=0.52 m−0.46 m), respectively. In the meantime, it can be seen that the error of a case where the angle from the optical axis of the camera is 50 degrees is increased to 0.51 m (=5.03 m−4.52 m) and 1.96 m (=6.48 m−4.52 m).
The influence by the error imposed upon the input image will be described by referring to FIG. 2. FIG. 2 is an illustration showing bird's-eye images 203, 204 created from images captured at continuous time series in a case where the actual camera attaching angle .theta. is shifted from the assumed angle by several degrees and a synthesized bird's-eye image 205 of those. For the bird's-eye images 203 and 204, the distance between white lines 201 becomes wider as the distance from the camera becomes farther, and diamond-like paint 202 that is a road marking indicating presence of a pedestrian crossing becomes extended in the longitudinal direction as the distance from the camera becomes farther.
When the captured image is converted into the bird's-eye image, it is converted on assumption that the camera attaching angle .theta. is the assumed angle. Thus, when the camera attaching angle .theta. is shifted by several degrees, distortion and position shift are generated in the converted bird's-eye image. The influence of the distortion and position shift becomes more extensive on the ground in the direction with larger angles with respect to the camera optical axis, i.e., in the positions farther from the own vehicle, so that the ways of distortions in the diamond-like paint 202 of the bird's-eye image 203 and the diamond-like paint 202 of the bird's-eye image 204 that is acquired when the own vehicle makes a move slightly therefore become different. Therefore, when the bird's-eye images 203 and 204 are simply superimposed on one another, obtained is an image in which positions of the diamond-like paints 202 and the white lines 201 do not match with each other as in the case of the synthesized bird's-eye image 205.
The technique disclosed in Patent Document 1 sets the conversion boundary line, and synthesizes only the section closer to the camera than to the conversion boundary line within the bird's-eye image to create the synthesized bird's-eye image. However, since the conversion boundary line setting method is not clearly defined, the visibility of the synthesized bird's-eye image becomes deteriorated by being influenced by the position shift and the distortion between the bird's-eye images when there is a camera parameter error.
Further, for the reflection of headlight of a following vehicle and the reflection of the streetlight reflected from the road surface, as shown in FIG. 3, there is often observed a road-surface reflection region 301 of the headlight of a following vehicle 302 in an area that is about 1 m from the own vehicle 303 in a captured image 300 that is taken at night. In such case, the road surface display on the bird's-eye image is hidden by the road-surface reflection of the headlight, so that the synthesized bird's-eye image comes to contain the influence of the road-surface reflection.
Further, for the lens extraneous matters such as raindrops and mud, as shown in FIG. 4, when a raindrop is attached to the lens bottom section, the bird's-eye image comes to contain the influence of a section 403 where a region 401 containing an image of the raindrop and a region 402 to be converted into a bird's-eye image overlap with each other within an image 400. Thus, a synthesized bird's-eye image containing the image of the raindrop is to be generated. In the case where there is mud attached within a range of the image to be converted into a bird's-eye image, a synthesized bird's-eye image containing the image of mud is to be generated as well.
Furthermore, for the shadow of the own vehicle, as shown in FIG. 5, the shadow of the own vehicle is observed as a low-brightness region 502 at an upper section of a bumper 501 of the own vehicle that is on a bottom section of a captured image 500. When this region 502 is converted into a bird's-eye image, the shadow of the own vehicle is captured within the bird's-eye image, thereby generating a synthesized bird's-eye image in which the shade and sunny sections are mixed.
When there is generated the synthesized bird's-eye image containing the distortion and noise due to the error of the camera parameter, misdetection and detection error may be caused when detecting a road marking on the road surface from such synthesized bird's-eye image.
It is therefore an object of the present invention to improve the issues of the conventional techniques described above and to provide a road marking image processing device which performs highly accurate road marking recognition by creating a high-quality synthesized bird's-eye image that has no influence of the distortion and position shift in the bird's-eye image caused by the camera parameter error and no influence of the reflection light from the road surface, lens extraneous matter, shadow of the own vehicle, and the like captured into the image.